Creating Generators Using yield in Python

Course: Python Training

Topic: Creating Generators Using yield

Course Link: Python Training Course – JustAcademy

Demo Registration: Register for a Course Demo

Website: JustAcademy Official Website

1. Introduction to Generators in Python

A generator is a special type of iterator in Python that produces values one at a time instead of creating and storing all values in memory at once. Generators are useful when working with large datasets, streams of information, repeated calculations, and data-processing tasks where values can be generated as needed.

The yield keyword is used to create a generator function. When Python encounters yield, it returns a value to the caller and temporarily suspends the function's execution. When the generator is resumed, execution continues from the point immediately after the yield statement.

Unlike a regular function that generally finishes when it reaches a return statement, a generator function can pause and resume multiple times, producing a sequence of values over its lifetime.

Learning Objectives

• Understand generators and their role in Python programming.
• Learn the difference between yield and return.
• Create generator functions and retrieve values from them.
• Use next() and for loops with generators.
• Understand lazy evaluation and memory efficiency.
• Apply generators to practical programming problems.
• Handle generator exhaustion and common errors.

2. What Is the yield Keyword?

The yield keyword is used inside a generator function to produce a value without permanently ending the function. After yielding a value, the function pauses and retains its local state, including local variables and its current execution position.

When the generator is requested to produce another value, it resumes from where it paused and continues executing until it encounters another yield, reaches the end of the function, or raises an exception.

Basic Syntax

def generator_function():
    yield value

A function containing yield becomes a generator function. Calling it returns a generator object rather than immediately executing the function body to completion.

Example: A Simple Generator

def simple_generator():
    yield 10
    yield 20
    yield 30

numbers = simple_generator()

print(next(numbers))
print(next(numbers))
print(next(numbers))

Output:

10
20
30

Explanation

1. simple_generator() creates a generator object.
2. The first next() call executes the function until yield 10 and returns 10.
3. The second call resumes execution and returns 20.
4. The third call resumes execution and returns 30.
5. After the final yielded value, the next request raises StopIteration because the generator has finished.

3. Understanding Generator Functions

A generator function looks similar to a regular Python function, but it uses yield to produce values over time.

Regular Function Example

def get_numbers():
    return [1, 2, 3]

result = get_numbers()
print(result)

Output:

[1, 2, 3]

This function creates and returns a complete list containing all three values.

Generator Function Example

def generate_numbers():
    yield 1
    yield 2
    yield 3

result = generate_numbers()
print(result)
print(next(result))
print(next(result))
print(next(result))

Output:

<generator object generate_numbers at ...>
1
2
3

The exact memory address shown in the generator object's representation varies. The important point is that calling the generator function returns a generator object, and its values are produced as the generator is consumed.

Key Characteristics of Generators

• They produce values one at a time.
• They support iteration.
• They preserve local variables and execution state while paused.
• They are evaluated lazily, meaning values are produced when requested.
• They are generally consumed once; after exhaustion, they do not restart automatically.
• They can reduce memory usage when compared with building a large list of all results.

4. Difference Between yield and return

Both yield and return can provide values from a function, but they have different effects on execution.

Example Comparing yield and return

def regular_function():
    return 1
    return 2

def generator_function():
    yield 1
    yield 2

print(regular_function())

for value in generator_function():
    print(value)

Output:

1
1
2

The second return in the regular function is never reached because the first return ends the function. In the generator function, each yield produces a value and suspends execution until the generator is resumed.

5. Creating a Generator with Multiple yield Statements

A generator can contain multiple yield statements. Each statement produces a value in the order in which execution reaches it.

Example: Generating Days of the Week

def days_of_week():
    yield "Monday"
    yield "Tuesday"
    yield "Wednesday"
    yield "Thursday"
    yield "Friday"

for day in days_of_week():
    print(day)

Output:

Monday
Tuesday
Wednesday
Thursday
Friday

How It Works

1. The generator begins when iteration requests its first value.
2. It yields Monday and pauses.
3. The loop requests another value, so execution resumes.
4. The process repeats for each remaining day.
5. When the function ends, iteration stops automatically.

This approach is useful when values are produced through a sequence of steps or when each result can be generated independently.

6. Using next() with Generators

The built-in next() function requests the next available value from an iterator, including a generator. Each call resumes the generator until it produces a value or finishes.

Example

def count_generator():
    yield 100
    yield 200
    yield 300

counter = count_generator()

print(next(counter))
print(next(counter))
print(next(counter))

Output:

100
200
300

Handling Generator Exhaustion

Calling next() after a generator has exhausted all its values raises StopIteration.

def one_value():
    yield "Hello"

items = one_value()

print(next(items))

try:
    print(next(items))
except StopIteration:
    print("Generator has no more values.")

Output:

Hello
Generator has no more values.

You can also provide a default value to next(). If the iterator is exhausted, Python returns the default instead of raising StopIteration.

def numbers():
    yield 5

items = numbers()

print(next(items, "Finished"))
print(next(items, "Finished"))

Output:

5
Finished

7. Using Generators with for Loops

A for loop is one of the simplest ways to consume a generator. Python repeatedly requests the next value and stops automatically when the generator is exhausted.

Example: Generate Numbers from 1 to 5

def generate_numbers():
    for number in range(1, 6):
        yield number

for value in generate_numbers():
    print(value)

Output:

1
2
3
4
5

Benefits of Using a for Loop

• It automatically requests values from the generator.
• It handles normal iterator exhaustion.
• It avoids manually calling next() for every value.
• It makes generator code easier to read and maintain.

8. Creating a Generator Using a Loop

Generators often use loops to create sequences dynamically. Instead of storing the complete sequence in a list, the function yields each value as needed.

Example: Generate Even Numbers

def even_numbers(limit):
    for number in range(0, limit + 1, 2):
        yield number

for value in even_numbers(10):
    print(value)

Output:

0
2
4
6
8
10

Example: Generate Squares

def square_generator(limit):
    for number in range(1, limit + 1):
        yield number ** 2

for square in square_generator(5):
    print(square)

Output:

1
4
9
16
25

These generators calculate each value when it is requested. They do not need to create a list containing every even number or square before iteration begins.

9. Understanding Lazy Evaluation

Lazy evaluation means that a value is calculated only when it is needed. Generators use lazy evaluation by producing values on demand rather than calculating the entire sequence in advance.

Example: Observe Generator Execution

def process_values():
    print("Starting generator")
    yield 10
    print("Preparing next value")
    yield 20
    print("Generator finished")

values = process_values()

print("Generator created")
print(next(values))
print(next(values))

Output:

Generator created
Starting generator
10
Preparing next value
20

The final message, Generator finished, is not printed because the program does not request another value after yielding 20. The generator has not yet executed the remaining statement in this example.

If another next(values) call were made, the generator would print Generator finished and then raise StopIteration.

Advantages of Lazy Evaluation

• Values are produced only when needed.
• Large sequences can be processed without storing every value in memory.
• Programs can begin processing data before all results have been generated.
• Generators can represent potentially unbounded sequences, provided consumers do not attempt to exhaust them completely.

10. Memory Efficiency: List vs Generator

Lists store their elements in memory, whereas generators can produce values incrementally. This difference is important when working with large ranges or data-processing pipelines.

Example Using a List

def get_squares_list(limit):
    return [number ** 2 for number in range(limit)]

squares = get_squares_list(5)
print(squares)

Output:

[0, 1, 4, 9, 16]

Example Using a Generator

def get_squares_generator(limit):
    for number in range(limit):
        yield number ** 2

squares = get_squares_generator(5)

for square in squares:
    print(square)

Output:

0
1
4
9
16

A generator is not automatically faster in every situation. If a small collection must be indexed repeatedly, a list may be more appropriate. Generators are especially useful when incremental processing or reduced memory use is important.

11. Creating an Infinite Generator

A generator can represent an infinite sequence by yielding values indefinitely. Such generators are useful for counters, repeated identifiers, and continuous value production. They should generally be consumed with a condition or a controlled number of iterations.

Example: Infinite Counter

def infinite_counter():
    number = 1

    while True:
        yield number
        number += 1

counter = infinite_counter()

for _ in range(5):
    print(next(counter))

Output:

1
2
3
4
5

Explanation

• The while True loop continues indefinitely.
• The generator yields the current number.
• After the generator resumes, it increments the number.
• The example requests only five values, so it does not loop forever in the consumer.

Important: Avoid converting an infinite generator into a list or iterating over it without a stopping condition, because the operation will not finish under normal circumstances.

12. Generator Expressions

Python supports generator expressions, which use syntax similar to list comprehensions but produce a generator object instead of a complete list.

List Comprehension

squares_list = [number ** 2 for number in range(1, 6)]
print(squares_list)

Output:

[1, 4, 9, 16, 25]

Generator Expression

squares_generator = (number ** 2 for number in range(1, 6))

for square in squares_generator:
    print(square)

Output:

1
4
9
16
25

Syntax Comparison

# List comprehension
[number ** 2 for number in range(1, 6)]

# Generator expression
(number ** 2 for number in range(1, 6))

The main syntax difference is the use of square brackets for a list comprehension and parentheses for a generator expression.

When to Use Generator Expressions

• When transforming values with a simple expression.
• When processing a sequence without storing all transformed results.
• When passing values into functions such as sum(), min(), or max().

Example: Sum of Squares

total = sum(number ** 2 for number in range(1, 6))
print(total)

Output:

55

13. Using yield to Process Large Datasets

Generators are useful for processing data in smaller portions. For example, a function can read a text file line by line instead of loading the entire file into memory.

Example: Read a File Line by Line

def read_file_lines(file_path):
    with open(file_path, "r", encoding="utf-8") as file:
        for line in file:
            yield line.rstrip("\n")

# Example usage:
for line in read_file_lines("sample.txt"):
    print(line)

Explanation:

• The file is opened when iteration first starts executing the generator body.
• The loop reads one line at a time.
• The generator yields the line without its trailing newline character.
• The file is closed when the loop completes normally or when the generator's context manager is exited as execution unwinds.

This pattern avoids reading the entire file into a list. If a consumer stops early and keeps the generator alive, the file may remain open until the generator is closed or cleaned up. For important resource-management requirements, explicitly close the generator or manage the file in a surrounding context manager.

Example: Generate Records

def generate_records():
    records = [
        {"id": 1, "name": "Asha"},
        {"id": 2, "name": "Ravi"},
        {"id": 3, "name": "Meera"}
    ]

    for record in records:
        yield record

for record in generate_records():
    print(record)

Output:

{'id': 1, 'name': 'Asha'}
{'id': 2, 'name': 'Ravi'}
{'id': 3, 'name': 'Meera'}

This example demonstrates incremental record production. Although the example's source data is already stored in a list, the same generator pattern can yield records from a database cursor, file, API response, or other streaming source.

14. Generator State and Resuming Execution

A generator preserves its local variables and execution position when it pauses at a yield statement. This makes it possible to build sequences that depend on previous calculations.

Example: Generator with State

def running_total():
    total = 0

    for number in range(1, 6):
        total += number
        yield total

for value in running_total():
    print(value)

Output:

1
3
6
10
15

Explanation

1. The variable total starts at 0.
2. The loop adds the current number to total.
3. The updated total is yielded.
4. The generator pauses while preserving the current value of total.
5. When resumed, the loop continues with the next number.

Generators can therefore maintain state without requiring the caller to manually store and pass the intermediate result back into the function.

15. Sending Values into a Generator with send()

Generators can also receive values from the caller using the send() method. A value sent into a suspended generator becomes the result of the current yield expression.

Example

def message_generator():
    message = yield "Generator is ready"
    yield f"Received: {message}"

generator = message_generator()

print(next(generator))
print(generator.send("Hello Python"))

Output:

Generator is ready
Received: Hello Python

How send() Works

• The first next() call starts the generator and pauses at the first yield.
• The call to send("Hello Python") resumes the generator.
• The sent string becomes the result of the suspended yield expression.
• The generator uses the value to create and yield the next message.

A non-None value generally cannot be sent to a generator that has not started yet. Start it first using next() or send(None).

16. Closing a Generator

The close() method requests that a generator stop. Python raises GeneratorExit inside the suspended generator. This is useful when a generator manages resources or when processing should stop before all values are produced.

Example

def sample_generator():
    try:
        yield 1
        yield 2
        yield 3
    finally:
        print("Generator cleanup completed")

generator = sample_generator()

print(next(generator))
generator.close()

Output:

1
Generator cleanup completed

The finally block executes during generator closure, allowing cleanup work to be performed. A generator should not yield another value while handling GeneratorExit.

17. Common Generator Errors and Troubleshooting

17.1 Using return Instead of yield

Problem: The function returns one value and exits instead of generating a sequence.

def wrong_generator():
    return 10
    return 20

Solution: Use yield when the function needs to produce multiple values over successive iterations.

17.2 Calling next() Too Many Times

Problem: Calling next() after the generator is exhausted raises StopIteration.

Solution: Consume the generator with a for loop or handle exhaustion with try and except StopIteration.

17.3 Expecting a Generator to Restart Automatically

Problem: A generator produces no values after it has been fully consumed.

Solution: Call the generator function again to create a new generator object.

def numbers():
    yield 1
    yield 2

first = numbers()
print(list(first))
print(list(first))

second = numbers()
print(list(second))

Output:

[1, 2]
[]
[1, 2]

17.4 Trying to Access a Generator by Index

Problem: A generator does not support direct indexing such as generator[0].

Solution: Iterate over the generator, use next() to retrieve the next item, or convert it into a list if all values can reasonably fit in memory.

17.5 Accidentally Creating an Infinite Loop

Problem: An infinite generator is consumed without a stopping condition.

Solution: Use a bounded loop, a condition, or a utility such as itertools.islice() to consume only the required number of values.

17.6 Assuming Generator Creation Executes the Body

Problem: Print statements inside the generator do not run when the generator object is created.

Solution: Remember that execution begins when iteration requests the first value, not when the generator function is called.

18. Best Practices for Using Generators

• Use yield when values can be generated one at a time.
• Choose generators for large datasets or streaming workflows when incremental processing is useful.
• Use for loops for normal generator consumption instead of repeated manual next() calls.
• Document whether a generator is finite or infinite.
• Validate inputs where necessary.
• Handle file handles, database cursors, and other resources carefully.
• Avoid converting large or infinite generators into lists without considering memory and termination.
• Use a list when repeated iteration, indexing, or slicing is required.
• Use generator expressions for short transformations that do not need a named generator function.
• Remember that generators are usually single-use iterators.
• Keep generator functions focused on producing a clear sequence of values.

19. Practical Mini Project: Generate an Even Number Sequence

In this mini project, create a generator that produces even numbers up to a specified limit. The program validates the limit and then prints each generated value.

def even_number_generator(limit):
    if limit < 0:
        raise ValueError("Limit must be non-negative")

    number = 0

    while number <= limit:
        yield number
        number += 2

try:
    limit = int(input("Enter the maximum limit: "))

    for number in even_number_generator(limit):
        print(number)

except ValueError as error:
    print("Invalid input:", error)

Example input:

Enter the maximum limit: 10

Output:

0
2
4
6
8
10

Project Explanation

1. The function receives a maximum limit.
2. It rejects negative limits.
3. The variable number begins at zero.
4. The while loop checks whether the current value is within the limit.
5. The yield statement produces the current even number.
6. After resuming, the generator increases the number by two.
7. The loop stops when the number exceeds the limit.

Possible Enhancements

• Generate odd numbers using a similar pattern.
• Allow the user to choose a starting number.
• Create a generator that produces multiples of a selected value.
• Filter generated values based on a condition.
• Write unit tests for zero, positive limits, and invalid inputs.

20. Practice Exercises

Exercise 1: Generate Numbers

Create a generator that yields the numbers 1 through 10.

Expected sequence: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Exercise 2: Generate Cubes

Create a generator that yields the cubes of numbers from 1 through 5.

Expected sequence: 1, 8, 27, 64, 125

Exercise 3: Generate Multiples

Write a generator that yields the first ten multiples of 3.

Expected sequence: 3, 6, 9, 12, 15, 18, 21, 24, 27, 30

Exercise 4: Read a Text File

Create a generator that reads a file one line at a time and yields each line without its trailing newline character.

Exercise 5: Infinite Counter

Create an infinite counter starting at 1 and use a loop to print only the first seven values.

Exercise 6: Compare List and Generator

Generate squares using both a list comprehension and a generator expression. Compare how values are retrieved and whether the results support indexing.

Exercise 7: Running Total

Create a generator that yields the cumulative total of numbers from 1 through 10.

Expected final total: 55

21. Interview Questions and Answers

Q1. What is a generator in Python?

Answer: A generator is an iterator that produces values one at a time, usually using a generator function with yield or a generator expression.

Q2. What is the purpose of yield?

Answer: The yield keyword produces a value and suspends execution so the generator can resume later.

Q3. What is the difference between yield and return?

Answer: return ends a function call, while yield pauses a generator and allows it to produce more values when resumed.

Q4. What is lazy evaluation?

Answer: Lazy evaluation means calculating values only when they are requested rather than calculating every value in advance.

Q5. How do you retrieve a value from a generator?

Answer: Use next() to retrieve one value or a for loop to iterate over its values.

Q6. What happens when a generator is exhausted?

Answer: It has no more values to produce. A direct next() call raises StopIteration, while a for loop ends automatically.

Q7. Can a generator be used more than once?

Answer: The same generator object cannot restart after exhaustion. Calling the generator function again creates a new generator object.

Q8. What is a generator expression?

Answer: A generator expression is a concise expression that creates a generator, using parentheses instead of the square brackets used for a list comprehension.

Q9. Why are generators useful for large files?

Answer: They can read and process data incrementally, avoiding the need to load an entire file into memory.

Q10. Do generators always use less memory than lists?

Answer: They often use less memory when values can be processed incrementally, but the exact benefit depends on the generator's implementation, retained state, and how the values are consumed.

Q11. Can a generator receive values?

Answer: Yes. The send() method can pass a value into a suspended generator, where it becomes the result of the current yield expression.

Q12. What does close() do?

Answer: It requests that the generator stop by raising GeneratorExit inside it, allowing cleanup logic such as a finally block to run.

22. Quick Revision Checklist

• A generator produces values one at a time.
• The yield keyword creates a generator function.
• A generator pauses at yield and resumes when requested.
• The generator function body begins executing when iteration starts.
• next() retrieves the next available value.
• A for loop consumes a generator automatically.
• StopIteration signals that no more values are available.
• Generator expressions use parentheses.
• Lazy evaluation can reduce memory usage.
• Generators are generally single-use iterators.
• send() can pass values into a suspended generator.
• close() can be used to stop a generator and trigger cleanup.
• Choose a list instead when indexing or repeated iteration is required.

23. Conclusion

Generators are an important Python feature for creating efficient and readable data-processing programs. By using yield, a function can produce a sequence of values over time, preserve its state, and resume execution whenever another value is requested.

Understanding generator functions, generator expressions, lazy evaluation, and iteration helps you write programs that handle large sequences and streaming data without unnecessarily storing every result in memory. Practise building number sequences, reading files line by line, and processing generated records to develop confidence with this concept.

Explore additional learning resources through these JustAcademy links:

• Explore the Python Training Course
• Register for a Course Demo
• Visit the JustAcademy Website